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Overview 


= Multi-scale metric & feedback loops 
e Design hazard analysis 
e Operational risk mitigation 
e Lifecycle discovery of surprises 





= Safety Performance Indicators (SPIs) 
e Beyond ‘vehicle acted unsafely” Atta 
e Beyond real-time dynamic risk measurement 
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e It's all about monitoring safety case validity 


© 2020 Philip Koopman 2 





Traditional Hazard Analysis € Fees 
= Risk Analysis (e.g., start with HARA) 

e List all applicable hazards 

e Characterize the resultant risk 

e Mitigate risk as needed Ettvine fap 

e Document all risks acceptably mitigated 
m Use various techniques to create hazard list 

e Lessons learned (previous projects; industry) 

e Brainstorming & analysis techniques 

— HAZOP, STPA, .... bring your own favorite approach ... 

= Limitation: unknown hazards 

e But, human is responsible for overall system safety 
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Hazard Analysis for ADAS 3 RESEARCH 
= Operating in the open world 
e All hazards arent known 
e New hazards will appear 
= Safety of the Intended Function (SOTIF) 
e Operate in the real world HAZARD 
ANALYSIS 
e Observe “triggering events” 
e Mitigate discovered hazards 
e Repeat 
= Limitation: unseen triggering events 
e But, human is responsible for system safety 













SOTIF 
TRIGGERING 
EVENTS 
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EDGE CASE 


Pre-Autonomy & ADAS Feedback Model G RESEARCH 


= Driver does dynamic risk mitigation 
= Recalls for technical faults 
e Recalls are never supposed to happen 





ie DRIVER 7 
HAZARD pe 
ANALYSIS { yy th abn a at | 
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Hazard Analysis for Full Autonomy G RESEARCH 


= Still an open world with unknowns & changes 


e But... 20 human driver responsible 


= Use Positive Trust Balance TRUSTWORTHY POSITIVE RISK BALANCE 
e Engineering rigor 














= 
e Practicable validation = = c = 
= O c- o) 
e Strong safety culture < a = i 
.. and ... a E 5 w 
e = LJ.| oc . i 
e Field feedback m ir = i 
to handle surprises Corse) 
Engineering Validation Safety 
Rigor Culture 
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Safety Arguments (Safety Case) 


= Claim — a property of the system 


e “System avoids pedestrians” CLAIM 


= Argument —- why this is true 





e “Detect & maneuver to avoid” SaaS 
m Evidence — supports argument ; : al 
e Tests, analysis, simulations, ... 
= Sub-claims/arguments address 
complexity 


e “Detects pedestrians’ // evidence 
e “Maneuvers around detected pedestrians’ // evidence 
e “Stops if can't maneuver’ // evidence 
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Default SDC Feedback Model 3 RESEARCH 


= Safety Case argues acceptable risk — without driver 
e Perhaps Positive Risk Balance (“safer than human”) 
e Update in response to incidents and loss events 
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TRIGGERING eh 


EVENTS LOSS EVENTS 


HAZARD 


e But, deployment only yields lagging metrics 
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Safety Performance Indicators (SPIs) (B'sescs 


= SPls monitor the validity of safety case claims 


LAGGING Vehicle is Safe ~ 
METRICS 


| CLAIMS-ONLY 
1 VIEW OF 
SAFETY CASE 
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Examples of SPls © RESEARCH 
= “Acts dangerously’ is only one dimension of SPls 
e Violation rate of pedestrian buffer zones 
e Time spent too close per RSS following distance 
= Components meet safety related requirements 
e False negative/positive detection rates 
e Correlated multi-sensor failure rates 
= Design & Lifecycle considerations 
e Design process quality defect rates 
e Maintenance & inspection defect rates 


= Is it relevant to safety? =} Safety Case =» SPls 
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KPI vs. SPI Contrast 


EDGE CASE 
RESEARCH 
=HOLOGRAM Distance to object: 





EXAMPLES OF TYPE e KPI: average and variance of clearance 


“STROLLER” 


e SPI: how often SDC violates safe clearance limit 


= Sensor effectiveness: 
e KPI: detection rate, SNR per sensor 
e SPI: concurrent multi-sensor detection failure 
e SPI: loss of calibration 
= Pedestrian perception: 
e KPI: accuracy, precision, recall 
e SPI: false negative more than <k> consecutive frames 
e SPI: systematic under-performance on sub-classes 
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Runtime Monitoring Implications 
= Responsibility-Sensitive Safety (RSS) Scenario: 
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e Safety monitor: increase distance if too close in case of panic stop 
e KPI: best effort separation given driving conditions 
e SPls: situation more dangerous than expected (e.g., ODD issues) 

— Spent more time in too-dense traffic than expected 

— Lead/own vehicle brake violate expectations 

— Other vehicles panic brake more often than assumed 
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SPlis and Lifecycle Feedback 


= SPI measures validity of a safety case claim 
= a SPI value violation means safety case is invalid 


lf 
Vn, 


= Root cause analysis might reveal: 
e Design process execution defect 
e Design defect 
e Hazard analysis gap 
e SOTIF analysis gap 
e Training data bias 
e Evidence gap, or defect 
e Assumption error 


SE i 
\ 






© 2020 Philip Koopman 13 





EDGE CASE 
RESEARCH 


SPI-Based Feedback Approach 


m Safety Case argues acceptable risk 
e SPIis monitor validity of safety case 


SOTIF 


NMOL TRIGGERING EVENTS 


RUN-TIME __ 
SAFETY “~~ 
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EDGE CASE 
Summary © RESEARCH 


= Monitoring incidents is only part of feedback 


= Removing human means mitigating surprise 
e Tactical: run-time safety monitoring 
e Strategic: run-time SPI monitoring 













Tier 1 
Suppliers 





AV 
Integrators 





= SPlis provide feedback on: 
e Design quality & process maturity 
e Testing coverage 
e Lifecycle procedure execution 

m= SPlIs: you are as safe as you think you are 
e Field feedback is key to SPI success 


Autonomy 
Stack 
Suppliers 


Rideshare 
Networks 






Fleet 
Operators 
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